Associating input/output device requests with memory associated with a logical partition

ABSTRACT

An address controller includes a bit selector that receives a first portion of a requester id and selects a bit from a vector that identifies whether a requesting function is an SR-IOV device or a standard PCIe device. The controller also includes a selector coupled to the bit selector that forms an output comprised of either a second portion of the RID or a first portion of the address portion based on an input received from the selector and an address control unit that receives the first portion of the RID and the output and determines the LPAR that owns the requesting function based thereon, the address control unit providing the corrected memory request to the memory.

BACKGROUND

The present invention relates to computing devices and, morespecifically, to systems and methods for associating input/output (I/O)requests with memory ranges assigned to particular logical partition(LPAR).

In a computing system, a logical partition (LPAR) is a subset ofcomputer's hardware resources, vitualized as a separate computer. Ineffect, a physical machine can be partitioned into multiple LPARs, eachsupporting a separate instance of an operating system.

Each LPAR may access memory from a common memory subsystem, providedthat the ranges of addresses directly accessible to each do not overlap.That is, each LPAR may be associated with a particular memory range.Special care is taken to ensure that that one LPAR cannot affect thememory assigned to another LPAR. One LPAR may, however, indirectlyaffect memory of a second partition, but only by commanding a process ofthe second partition to directly operate on its memory.

At any given time, an LPAR may be associated with one or more PCIedevices (I/O cards). Typically, multiple I/O cards are coupled to PCIeswitches and the LPAR communicates to the individual cards through thePCIe switch.

In some cases, an I/O card or other PCIe device may comply with theSingle Root I/O Virtualization (SR-IOV) specification. The SR-IOVspecification allows a single PCIe device to represent multiple virtualdevices where each virtual device appears to be a separate physical PCIedevice. Thus, in the server context, a single I/O card conforming to theSR-IOV specification may be partitioned into multiple virtual devices.Such PCIe devices shall be referred to herein as “SR-IOV devices.” Inother cases, the PCIe devices may only represent one device and arereferred to herein as “standard PCIe devices.”

In operation, when an LPAR is controlling a PCIe device that device mayneed to read from or write to the memory associated with the LPAR. It isimportant that the PCIe device only have access to the memory rangeassociated with the LPAR controlling it. Otherwise, one LPAR mayindirectly affect the memory range assigned to another LPAR thoughmemory access operations performed by a PCIe device.

In the case of SR-IOV compliant devices, a function number portion ofthe requestor ID (RID) identifies the particular virtual device. Thisfunction number may be associated with a particular LPAR and the LPARmay instruct the virtual device to use any address in the PCI addressspace. In the case of a non-SR-IOV compliant devices such a functionnumber is not provided, and firmware may ensure that the addresses inrequests made by the device are located within the memory range for theLPAR.

The difference in memory address request formats has required that eachPCIe Root Complex be connected to only one of the two types of PCIedevices. In some instances, however, it may be desirable to have bothSR-IOV compliant devices and non-SR-IOV devices connected to a singleRoot Complex through one or more PCIe switches. Current systems may notadequately support such a connection while ensuring PCIe device memoryaccess requests are contained in the memory assigned to the LPARcontrolling the PCIe device.

SUMMARY

According to one embodiment of the present invention, an addresscontroller that receives a memory access request from a requestingfunction, the request including an address portion and a requesteridentification (RID) and provides a corrected memory request to a memorythat does not request access to a portion of the memory not assigned toa logical partition LPAR that owns the requesting function. The addresscontroller of this embodiment includes a bit selector that receives afirst portion of the RID and selects a bit from a vector that identifieswhether the requesting function is an SR-IOV device or a standard PCIedevice. The address controller also includes a selector coupled to thebit selector that forms an output comprised of either a second portionof the RID or a first portion of the address portion based on an inputreceived from the selector. In addition, the address controller includesan address control unit that receives the first portion of the RID andthe output and determines the LPAR that owns the requesting functionbased thereon, the address control unit providing the corrected memoryrequest to the memory.

Another embodiment of the present invention is directed to a computingsystem comprising that includes a host computing device. The hostcomputing device includes a memory having multiple ranges and aprocessor divided into two or more logical partitions, each logicalpartition being associated with a one of the multiple ranges. The systemalso includes an input/output hub coupled to the host computing device,a PCIe expansion card coupled to input/output hub and two or more I/Odevices coupled to the PCIe expansion card, including an SR-IOV deviceand a standard PCIe device. In this embodiment, the input/output hubreceives a memory access request from a one of the two or more I/Odevices through the PCI expansion card and provides a corrected memoryrequest to a memory that does not request access to a portion of thememory not assigned to a logical partition LPAR that owns the one of theI/O devices.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention. For a better understanding of the invention with theadvantages and the features, refer to the description and to thedrawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The forgoing and other features, and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIG. 1 shows one embodiment of a computing system according to oneembodiment of the present invention;

FIG. 2 shows an example of an address controller according to oneembodiment of the present invention; and

FIG. 3 shows an address control unit that is part of the addresscontroller of FIG. 2 according to one embodiment.

DETAILED DESCRIPTION

Embodiments of the present invention may help ensure that a PCIefunction memory access request does not access memory assigned to anyLPAR other than the LPAR owning the function. This may be accomplishedin one embodiment with a single circuit that may operate on requestsfrom both SR-IOV devices and standard PCIe devices. As the term is usedherein, a “function” refers to a particular PCIe device, whether avirtual device or standard device. Accordingly, an SR-IOV device mayinclude one or more functions and a standard PCIe device includes onlyone function.

It will be assumed that when an LPAR is given ownership of a function,that ownership association is stored in some location, typically asdevice table. The association may be used to produce a Zone_ID used toindex into a zone relocation mechanism to convert a zone absoluteaddress into a system absolute address. The Zone_ID may be determinedbased on information contained in the PCIe request header.

As two different types of PCIe devices may exist, according to oneembodiment, two different ways of determining the Zone_ID may beprovided. In one embodiment, a single circuit may be utilized todetermine the Zone_ID regardless of the type of PCIe device making therequest. In the case of SR-IOV devices, the function may be identifiedby a portion of the PCI Requester ID (RID) field. In this case, the RIDis used to determine the Zone_ID. In the case of standard PCIe devicesthere is only one function and no virtual functions. Thus, all RIDfields are the same for all the LPARs and include only a bus number.Accordingly the bus number is supplemented with a portion of the PCIeaddress field to uniquely identify the requester and serve as the indexinto a table containing the Zone_ID associations.

FIG. 1 shows one embodiment of a computing system 100 according to oneembodiment of the present invention. The computing system 100 includes ahost computing device 102. It will be appreciated that the hostcomputing device 102 can be any suitable computer or computing platform,and may include a terminal, wireless device, information appliance,device, workstation, mini-computer, mainframe computer, personal digitalassistant (PDA) or other computing device. It shall be understood thatthe device 102 may include multiple computing devices linked together bya communication network. For example, there may exist a client-serverrelationship between two systems and processing may be split between thetwo.

The host computing device 102 may be divided into one or more logicalpartitions (LPARs). As shown, the host computing device 102 is dividedinto three LPARs, a first LPAR 104, a second LPAR 106 and a third LPAR108. Of course, any number of LPARs could be formed.

Each LPAR 104, 106, and 108 may include its own operating system and runapplications independent of one another. Examples of operating systemsthat may be supported by the LPARs 104, 106 and 108 include zOS, Windows95, Windows 98, Windows NT 4.0, Windows XP, Windows 2000, Windows CE,Windows Vista, Mac OS, Java, AIX, LINUX, and UNIX, or any other suitableoperating system.

The host computing device 102 may include memory 110. The memory may bedivided into several ranges. These ranges may be assigned to individualLPARs. In the example shown in FIG. 1, the first LPAR 104 is assigned toa first memory range 111, the second LPAR 106 is assigned to a secondmemory 112, and the third LPAR 108 is assigned to a third memory range113.

In addition, the host computing device 102 may include a functionality116 that ensures that each LPAR only accesses memory assigned to it.This functionality 116 may include trusted firmware for assigningaddresses for memory blocks and for controlling certain memory accessrequests for one or more functions. The functionality unit 116 may beassigned to a fourth memory range 114. The functionality unit 116 mayoperate in different manners depending on the type of adapter beingaccessed. For example, for an SR-IOV adapter, firmware in thefunctionality unit 116 configures the I/O hub 120 (discussed below) andthen the owing LPAR is free to create control blocks containing theaddresses to be used by the SR-IOV adapter. In contrast, for standardadapters, the functionality unit 116 receives requests from the owingLPAR and creates the control blocks. As such, the functionality unit 116ensures that the addresses are within the range owned by the LPAR.

In operation, memory 110 is controlled by a memory controller 118. Thememory controller 118 receives memory access requests from multiplelocations. In one embodiment, the memory controller 118 may receiverequests from one or more functions.

The system 100 may also include an I/O hub 120. The I/O hub 120facilitates communication between the memory 110 and one or morefunctions. The I/O hub 120 is coupled to the host computing device 102by an I/O bus 122. The I/O hub 120 may be included within the hostcomputing device 102 in some embodiments.

The I/O hub 120 may include one or more address controllers according toone embodiment of the present invention. As shown if FIG. 1, the I/O hub120 includes first address controller 122, a second address controller124, a first root complex 123, and a second root complex 125. The firstroot complex 123 and the second root complex 125 include thefunctionality required to send and receive PCIe packets. Thisfunctionality may include header, link, and transaction processing. Thefirst address controller 122 is coupled to a first PCIe expansion device126 via a first PCIe connection 127. In one embodiment, the first PCIeexpansion device 126 is located in an I/O drawer that includes multipleslots. The PCIe expansion device 126 may be coupled via a backplane toother slots in the I/O drawer. For ease of explanation, the connectionis given what is referred to herein as a bus number. Accordingly, thebus number may refer to a particular slot a PCIe device is located.

In the example shown in FIG. 1, the first PCIe expansion device 126 iscoupled by a first PCIe bus 132 to a first PCIe device 128. The firstPCIe device 128, in this example, may be an SR-IOV device. That is, thefirst PCIe device 128 may support multiple functions. The first PCIeexpansion device 126 may also be coupled by a second PCIe bus 134 to asecond PCIe device 130. In this example, the second PCIe device 130 maya standard PCIe device. That is, it may only support a single function.

In addition, the second address controller 124 may be coupled to asecond PCIe expansion device 131 through a second PCIe connection 133.The second PCIe expansion device 131 may be coupled to a third PCIedevice 134 via a third PCIe bus 135 in the same or similar manner asdescribed above.

In operation, the address controllers 122 and 124 receive memory accessrequests from one or more functions of the PCIe devices 128, 130 and134. Each of these requests may include PCI Requester ID (RID) and anaddress field. The address field may be 64 bits wide in one embodiment.

Either the RID alone or a combination of the RID and a portion of theaddress, depending on the type of PCIe device making the request, may beutilized by the address controllers 122 and 124 to determine which LPARowns the function making the request. Based on which LPAR owns thedevice, the address controllers 122 and 124 ensure that the address iswithin the memory range assigned to the LPAR that owns the function.

FIG. 2 shows an example of an address controller 200 according to oneembodiment of the present invention. The address controller 200 receivesa memory access request 202 from a function and outputs a real memoryaddress 204 that is provided to the memory controller 118 (FIG. 1).

The address controller 200 includes a selection portion 206 that, basedon the access request 202, creates an index 208 used by an addresscontrol unit 211 to identify the LPAR controlling the requestingfunction. The index 208 is created by selecting a first portion (shownas bits 0-7 in FIG. 2) of the RID 220 and combining them with either theremainder of the RID (in the case that an SR-IOV device is making therequest) or with a portion 210 of the address portion 213 of the request202 (in the case that a standard PCI device is making the request). Inone embodiment, the portion 210 may be 8 bits wide.

In more detail the selection portion 206 includes a bit selector 212.The bit selector 212 receives the first portion of the RID. The bits ofthe first portion identify the bus (slot) number of the requestingfunction. This number is used by the bit selector 212 to select a bitfrom a vector 214. In one embodiment, the vector 214 may be created byeither configuration firmware or an autodetect function that determinesthe type of PCIe card in each slot of an I/O drawer. The vector 214 mayinclude either a 1 or zero in each bit where each bit corresponds to anindividual PCIe bus number (i.e., slot in an I/O drawer).

The selected bit indicates whether the requesting device is an SR-IOVdevice or a standard PCIe device. For example, and as illustrated inFIG. 2, the vector 214 may store a zero to indicate that the device isan SR-IOV device and a 1 to indicate that it is a standard PCIe device.

In the case of SR-IOV compliant devices, the remainder (shown as bits8-15 in FIG. 2) of the RID identify a particular virtual device. In thecase of standard PCIe device, these bits are always zeros. The bitselector 212 may be coupled to a multiplexer 230 or other selectiondevice that selects either the remainder of the RID or the portion 210of the address 202 to produce the index 208. In the case whererequesting device is a standard PCIe device the device is instructed, bya control block, what values are to be placed in the portion 210. Inparticular, the values in portion 210 may be assigned by thefunctionality block 116 (FIG. 1) and stored in a table for later use.

The index 208 is provided to the address control unit 211. The addresscontrol unit 211 may also receive at least a portion of the address 213.In one embodiment, the address control unit 211 receives 48 bits (e.g.,bits 16-63) of the address 213 (in the case where the address is 64 bitslong) and creates real memory address 208 therefrom in such a mannerthat the real address is located in the memory range assigned to theLPAR owing the requesting function. In one embodiment, the some of thebits (e.g., bits 52-63) of the address 213 are passed directly throughthe address control unit 211 because they may not be able to effectwhich memory range is selected because the ranges be greater than 1megabyte. Of course, depending on the zone sizes, a different number ofbits may not be altered.

FIG. 3 shows an address control unit 211 according to one embodiment. Inthis embodiment, the address correction includes a look-up table (LUT)302. The LUT 302 includes an index to an entry in the device table 304that in turn includes a record of which LPAR owns a particular functionat a particular time. The LUT 302 is accessed by the index 208 whichincludes the bus number (card slot) and the function number or anassigned number of the requesting function. In one embodiment, the LUTis formed by a content addressable memory (CAM).

LUT 302 creates an output that is the index into a device table 304. Thedevice table 304, based on the input, creates an LPAR identification ofthe LPAR owning the requesting function. The LPAR identification is anindex into another table, the LPAR relocation table 306 that includes abase and size of a memory associated with each LPAR. Depending on theLPAR identified by the device table 304, the LPAR relocation table 306outputs a base 308 and size output 310. The base output 308 identifiesthe real memory address of a particular range and the size output 310indicates the real address of the upper limit of the range.

The address control unit 211 may also include an address translation andprotection unit 312. This unit 211 converts a requested address to areal address in memory. The requesting function does not, however, knowthe range of memory assigned to the LPAR that owns it. Thus, the addressis added to the base portion 308 by adder 314 to create a real memoryaddress 204. However, in the event that the requested address is greaterthan a size of the memory range assigned to the LPAR as determined bycomparator 316, an a error may be generated.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneore more other features, integers, steps, operations, elementcomponents, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

The flow diagrams depicted herein are just one example. There may bemany variations to this diagram or the steps (or operations) describedtherein without departing from the spirit of the invention. Forinstance, the steps may be performed in a differing order or steps maybe added, deleted or modified. All of these variations are considered apart of the claimed invention.

While the preferred embodiment to the invention had been described, itwill be understood that those skilled in the art, both now and in thefuture, may make various improvements and enhancements which fall withinthe scope of the claims which follow. These claims should be construedto maintain the proper protection for the invention first described.

What is claimed is:
 1. An address controller that receives a memoryaccess request from a requesting function, the request including anaddress portion and a requester identification (RID) and provides acorrected memory request to a memory, wherein the correct memory requestdoes not request access to a portion of the memory not assigned to alogical partition LPAR that owns the requesting function, the addresscontroller comprising: a bit selector that receives a first portion ofthe RID and selects a bit from a vector that identifies whether therequesting function is an SR-IOV device or a standard PCIe device; aselector coupled to the bit selector that forms an output comprised ofeither a second portion of the RID or a first portion of the addressportion based on an input received from the selector; and an addresscontrol unit that receives the first portion of the RID and the outputand determines the LPAR that owns the requesting function based thereon,the address control unit providing the corrected memory request to thememory.
 2. The address controller of claim 1, wherein the addresscontroller is coupled to one or more PCIe expansion cards.
 3. Theaddress controller of claim 2, wherein at least one of the PCIeexpansion cards is coupled to at least one SR-IOV device and at leastone standard PCIe device.
 4. The address controller of claim 1, whereinthe corrected memory request includes an address corresponding to a realmemory location in the memory.
 5. The address controller of claim 1,wherein the address control unit includes a content addressable memorythat determines the LPAR that owns the function.
 6. The addresscontroller of claim 5, wherein the address control unit includes adevice table and a relocation table coupled to the device table.
 7. Theaddress controller of claim 6, wherein the device table receives aninput from the content addressable memory and provides an output to therelocation table causing the relocation table to output the base andsize for the LPAR that owns the function.
 8. The address controller ofclaim 6, wherein the address control unit includes an adder that adds asecond portion of the address to the base to create an address portionof the corrected memory request.
 9. A computing system comprising: ahost computing device including: a memory having multiple ranges; aprocessor divided into two or more logical partitions, each logicalpartition being associated with a one of the multiple ranges; aninput/output hub coupled to the host computing device; a PCIe expansioncard coupled to input/output hub; two or more I/O devices coupled to thePCIe expansion card, including an SR-IOV device and a standard PCIedevice; wherein the input/output hub receives a memory access requestfrom a one of the two or more I/O devices through the PCI expansion cardand provides a corrected memory request to a memory, wherein the correctmemory request does not request access to a portion of the memory notassigned to a logical partition LPAR that owns the one of the I/Odevices.
 10. The computing system of claim 9, wherein the I/O hubincludes a bit selector, the bit selector receiving a first portion of arequester identification (RID) contained in the memory access requestand selects a bit from a vector that identifies whether the requestingI/O device is an SR-IOV device or a standard PCIe device.
 11. Thecomputing system of claim 10, wherein the I/O includes a selectorcoupled to the bit selector that forms an output comprised of either asecond portion of the RID or a first portion of an address portion ofthe memory request based on an input received from the selector.
 12. Thecomputing system of claim 9, wherein the corrected memory requestincludes an address corresponding to a real memory location in thememory.
 13. The computing system of claim 9, wherein the input/outputhub includes an address control unit coupled an output of the selector.14. The computing system of claim 13, wherein the address control unitincludes a content addressable memory that determines the LPAR that ownsthe requesting I/O device.
 15. The computing system of claim 14, whereinthe wherein the address control unit includes a device table and arelocation table coupled to the device table.
 16. The computing systemof claim 15, wherein the device table receives an input from the contentaddressable memory and provides an output to the relocation tablecausing the relocation table to output the base and size for the LPARthat owns the function.
 17. The address controller of claim 16, whereinthe address control unit includes an adder that adds a second portion ofthe address portion of the memory request to the base to create anaddress portion of the corrected memory request.